The village of Castel San Giovanni stands on a rocky crag at the far southern end of the valley, Valle di Castello.

The high ground appears to have been artificially levelled off to form a sort of rectangular platform, with steep, rocky walls forming a natural defence on three sides. A road lined with houses crosses the centre of the platform. The village has two separate nuclei on different levels separated by a saddle.

The castle that gives it its place name stood on the highest part of the hill but its transformation into a settlement which took place in the 16th century erased all signs of fortification. An older settlement (the village?) was probably sited in the second nucleus of Castel S. Giovanni: the normal layout of the later houses seems to have followed that of a pre-existing building; an interesting sixteenth century building with its original, elegantly carved, stone portal stands out from the village buildings; the majority of its windows, which were once decorated with stucco monsters, have been boarded up, a sign of the state of neglect in which the building lies.

To the east of the village at an altitude of 1,030 m. a.s.l. is a circular, 50 m in diameter enclosure with terraces sloping down towards Fustagna. On the north-eastern edge of the enclosure lies a circular millstone measuring approximately 15 metres in diameter, another similar stone can be found a little further north at an altitude of 1,040 m .a.s.l. All around is an area of fictile fragments.